{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第七层以外的实现方式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Level 7 or Level 4: What if we dig deeper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Layer | Name        | Protocol    | Device       | Code                     |\n",
    "|-------|-------------|-------------|--------------|--------------------------|\n",
    "| 7     | Application | eg. HTTP    | Load balancer| User land*, eg. nginx    |\n",
    "| 4     | Transport   | TCP UDP     | NAT          | Kernel*                  |\n",
    "| 3     | Network     | IP          | Router       | Kernel                   |\n",
    "| 2     | Data link   | Ethernet    | Switch       | Driver/Hardware          |\n",
    "| 1     | Physical    |             | Hub          | Hardware                 |\n",
    "\n",
    "\n",
    "- **\\*Exceptions:** TCP offload engine, http.sys on Windows, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> https://www.zhihu.com/question/27668104/answer/37622036 <br/>\n",
    "> 我来个简单粗暴文不对题的：\n",
    "> - 连MAC都不知道的算第1层，例如已经死绝了的hub\n",
    "> - 只知道MAC不知道[P的算第2层，例如普通交换机\n",
    "> - 只知道IP不知道port也就不管TCP还是UDP的算第3层，例如普通路由器\n",
    "> - 知道IP还知道port的算第4层，例如 NAT\n",
    "> - 关心payload的算第7层，例如 http proxy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OSI模型协议层次划分标准"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 分层标准争议：协议归属层级的划分存在不同观点，例如ARP协议在《图解TCP/IP》中被归为网络层，而维基百科将其归为数据链路层\n",
    "- 典型分层结构：\n",
    "  - 第7层应用层：HTTP协议，实现设备如负载均衡器（如nginx），代码通常在用户态\n",
    "  - 第4层传输层：TCP/UDP协议，NAT设备工作于此层，代码在内核态实现\n",
    "  - 第3层网络层：IP协议，路由器工作于此层\n",
    "  - 第2层数据链路层：以太网协议，交换机工作于此层\n",
    "  - 第1层物理层：物理连接设备如Hub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## reuse port\n",
    "\n",
    "- 实现差异：\n",
    "  - Windows系统通过http.sys内核服务实现多进程共享同一端口服务不同URL路径\n",
    "  - Linux/Unix系统通过reuse port选项实现多进程共享监听端口，但无法按URL路径分发\n",
    "- 技术背景：端口共享需要内核态支持，Windows XP SP2开始在内核实现HTTP服务"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Level 7: TCP relays we have seen so far"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For $N$ clients, TCP relay manages $2*N$ connections\n",
    "  - Each TCP connection uses at least 2KB physical memory in kernel\n",
    "    - `sizeof(struct tcp_sock)` alone, not counting others"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![9.1](./images/9.1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- $2*N$ TCP state machines: retransmission, congestion control, etc.\n",
    "- Why not pass TCP segments through?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 资源消耗：\n",
    "  - 每个TCP连接需要占用﻿$>2k$﻿物理内存（TCP_socket结构体约1800字节）\n",
    "  - 实现TCP relay需要﻿$2n$﻿个连接（n个客户端），内存消耗约﻿$4k$﻿每客户端\n",
    "- 性能瓶颈：\n",
    "  - 数据需要在用户态和内核态间多次拷贝（read/write操作）\n",
    "  - 需维护完整的TCP状态机，包括重传和拥塞控制\n",
    "- 优化方案：Linux的splice系统调用可减少内存拷贝，但使用限制较多"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NAT approach for TCP relay is simpler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![9.2](./images/9.2.png)\n",
    "\n",
    "- Much less memory and CPU usage, say 100B per client  \n",
    "- FYI: Run TCP/IP stack in user land, ref. [www.wanproxy.org](http://www.wanproxy.org)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 工作原理：\n",
    "  - 在TCP segment级别进行网络地址转换（修改IP和port）\n",
    "  - 不维护完整TCP状态，重传由终端设备自行处理\n",
    "- 性能优势：\n",
    "  - 内存消耗降至约100字节/session\n",
    "  - 免去拥塞控制等复杂逻辑，CPU使用率显著降低\n",
    "- 替代方案：用户态TCP协议栈（如基于FreeBSD内核协议栈改造）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 谷歌Maglev网络负载均衡器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- https://research.google/pubs/maglev-a-fast-and-reliable-software-network-load-balancer/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 设计特点：\n",
    "  - 软件实现的第四层负载均衡器，运行于普通Linux服务器\n",
    "  - 通过ECMP（等价多路径路由）实现流量分发\n",
    "  - 单机可处理10Gbps小包流量\n",
    "- 技术优势：\n",
    "  - 成本仅为硬件方案的1/5，性能达硬件方案的1/3\n",
    "  - 支持一致性哈希和连接跟踪，保证故障时连接不中断\n",
    "- 实现细节：\n",
    "  - 使用Virtual IP(VIP)技术实现单IP多服务\n",
    "  - TCP MSS值设置小于标准1460字节，适应软件处理特性"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 正确理解TCP的可靠性"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 特性对比:\n",
    "  - TCP：面向连接的、可靠的字节流协议\n",
    "  - UDP：无连接的、不可靠的数据报协议\n",
    "- 可靠性定义:\n",
    "  - 实现\"不重不漏不乱\"的数据传输\n",
    "    - 不重：数据不会重复发送\n",
    "    - 不漏：数据不会丢失\n",
    "    - 不乱：数据顺序不会错乱\n",
    "  - 但不保证数据内容不被修改"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 知乎帖子答疑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.zhihu.com/question/25016042/answer/29798924\n",
    "\n",
    "\n",
    "- Q\n",
    "```\n",
    "找了好多答案，都是只能从应用层发送一个确认。\n",
    "TCP已经使用一些机制可靠的发送过去，为何传输层不向上提供一个“ACK”，还要让开发者从应用层发送一个“ACK\",这样TCP不是更慢了吗?\n",
    "另外UDP的Send recv又怎么判断对方是否成功接收?\n",
    "不知道问得准不准确，如果有不对的地方，请大家指正。\n",
    "```\n",
    "\n",
    "\n",
    "- A \n",
    "```\n",
    "你问得很准确，答案也很明确，就是要应用层发送一个确认。\n",
    "TCP 的 ACK 表示对方的协议栈已经收到了你发的数据，不代表对方的应用程序收到了你发的消息。\n",
    "对方的应用程序可能死锁或者阻塞，不会去调用 recv()，那么你发的数据就堆积在对方协议栈的接收缓冲区里了。\n",
    "这种情况你认为算是对方收到还是没有收到消息呢？\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TCP is reliable, but not infallible"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Effective TCP/IP Programming, Tip 9\n",
    "\n",
    "- What does `send(sockfd, \"hello\", 5)` return `5` really mean?\n",
    "  - TCP/IP stack of localhost got your data, that's it, period.\n",
    "  - No guarantee that the remote program willever get it, unchanged\n",
    "  - Client can send data and get `ACK` before server `accept()`\n",
    "- Naïve to write following code and hope the TCP will do the magic\n",
    "  -   connect, write, close \n",
    "  - It will work intest env., and eventually break in production some day\n",
    "- Strongly suggest putting a *checksum* in your message (§A.1.13)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `send`返回值本质:\n",
    "  - 仅表示本地TCP/IP协议栈已接收数据\n",
    "  - 不保证数据已发送到网络或对方已接收\n",
    "- 应用层确认必要性:\n",
    "  - TCP的ACK仅表示对方协议栈收到数据\n",
    "  - 对方应用程序可能因死锁/阻塞无法处理数据\n",
    "  - 真实案例：数据可能堆积在接收缓冲区未被处理\n",
    "- 编程建议:\n",
    "  - 关键消息应包含校验和(checksum)\n",
    "  - 不能仅依赖TCP头部校验和\n",
    "  - 参考书籍附录A.1.13节详细说明\n",
    "\n",
    "- 常见误区:\n",
    "  - 天真地认为TCP能完全保证数据可靠传输\n",
    "  - 测试环境表现良好不代表生产环境可靠\n",
    "  - 简单网络环境(如直连交换机)可能掩盖问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Partial failure (§9.1.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In a distributed system, the failure of a network link is indistinguishable from the failureof a processor on the other side of that link. (A Note on Distributed Computing by Jim Waldo et al)\n",
    "- Fallacies of distributed computing (wiki)\n",
    "  - 1 of 8: The network is reliable\n",
    "- Designing Distributed Systems by Ken Arnold\n",
    "  - http://www.artima.com/intv/distrib.html\n",
    "- Introduction to Distributed System Design by Google\n",
    "  - http://www.hpcs.cs.tsukuba.ac.jp/~tatebe/lecture/h23/dsys/dsd-tutorial.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 核心概念:\n",
    "  - 网络链路故障与远端处理器故障无法区分\n",
    "  - 分布式计算八大谬误之首：认为网络可靠\n",
    "- 典型场景:\n",
    "  - 发送消息后网络故障的两种可能：\n",
    "    - 消息已送达但响应丢失\n",
    "    - 消息未送达\n",
    "  - 发送方无法确定实际发生的情况\n",
    "- 设计启示:\n",
    "  - 必须假设故障随时可能发生\n",
    "  - 放弃本地/远程透明性的幻想\n",
    "  - 保持系统简单以降低恢复复杂度\n",
    "  - 限制可能的恢复策略数量\n",
    "- 推荐阅读:\n",
    "  - [A Note on Distributed Computing by Jim Waldo](https://waldo.scholars.harvard.edu/publications/note-distributed-computing)\n",
    "  - [Ken Arnold关于分布式系统的访谈](https://www.artima.com/articles/designing-distributed-systems)\n",
    "  - [Google分布式系统设计导论(存档版)](https://www.hpcs.cs.tsukuba.ac.jp/~tatebe/lecture/h23/dsys/dsd-tutorial.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Muduo与C++11/14/17"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Muduo has **cpp11** branch since Feb 2014\n",
    "- Muduo **master branch (C++98)** supports **gcc 4.4** and onwards  \n",
    "  - **Debian 6**, **Ubuntu 10.04**, **RedHat 6** all have **gcc 4.4** and kernel **2.6.32**\n",
    "- How about C++11?\n",
    "\n",
    "| Linux distros        | Kernel | gcc version |\n",
    "|-----------------------|--------|-------------|\n",
    "| Ubuntu 12.04         | 3.2    | 4.6         |\n",
    "| Debian 7 (2013-05)   | 3.2    | 4.7         |\n",
    "| Ubuntu 14.04         | 3.13   | 4.8         |\n",
    "| RedHat 7 (2014-06)   | 3.10   | 4.8         |\n",
    "| Debian 8 (2015-04)   | 3.16   | 4.9         |\n",
    "| Ubuntu 16.04         | 4.4    | 5.3         |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 分支历史: 木多从2014年2月开始在GitHub上提供cpp11分支，支持C++11特性\n",
    "- 主分支特性: 主分支(master)基于C++98/C++03标准，支持gcc 4.4及以上版本\n",
    "- 兼容性考虑: 选择gcc 4.4是因为2010年主流发行版(Debian 6/Ubuntu 10.04/RedHat 6)都默认使用该版本和2.6.32内核"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "C++11\n",
    "- 发行版支持:\n",
    "  - Ubuntu 12.04: 内核3.2 + gcc 4.6\n",
    "  - Debian 7(2013-05): 内核3.2 + gcc 4.7\n",
    "  - Ubuntu 14.04/RedHat 7: 内核3.13/3.10 + gcc 4.8\n",
    "  - Debian 8(2015-04): 内核3.16 + gcc 4.9\n",
    "  - Ubuntu 16.04: 内核4.4 + gcc 5.3\n",
    "- 版本选择建议: gcc 4.7开始正式支持`-std=c++11`选项，建议作为C++11支持的最低版本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## All boost dependencies will be removed in C++17"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Boost 库/特性                  | 对应的标准库替代                         | 说明                         |\n",
    "|--------------------------------|------------------------------------------|------------------------------|\n",
    "| `boost::shared_ptr` / `weak_ptr` | `std::shared_ptr` / `std::weak_ptr`     | 智能指针                     |\n",
    "| `boost::function` / `bind`     | `std::function` / `std::bind` / lambda   | 函数对象与绑定               |\n",
    "| `boost::scoped_ptr`            | `std::unique_ptr`                        | 独占智能指针                 |\n",
    "| `boost::ptr_vector<T>`         | `std::vector<std::unique_ptr<T>>`        | 容器管理智能指针             |\n",
    "| `BOOST_STATIC_ASSERT`          | `static_assert`                          | 编译期断言                   |\n",
    "| `<boost/type_traits/xxx.hpp>`  | `<type_traits>`                          | 类型萃取                     |\n",
    "| `boost::any`                   | `std::any` *(C++17, gcc7+)*              | 类型安全存储任意值           |\n",
    "|                                | `std::experimental::any` *(gcc5)*        | 提前实验性支持               |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimal gcc version for muduo in C++11?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **gcc5** has a new **ABI**, e.g. `std::list<T>::size()` is `O(1)` now\n",
    "- **gcc5** changed `std::string` to **SSO** (short string optimization)\n",
    "  - No need to `typedef muduo::string` as `__gnu_cxx::__sso_string`\n",
    "  - Works better with **Protobuf** and other libraries\n",
    "\n",
    "\n",
    "| Linux distros                | gcc version                  |\n",
    "|-------------------------------|------------------------------|\n",
    "| Ubuntu 12.04                 | 4.6                          |\n",
    "| Debian 7 (2013-05)           | 4.7 // accepts `-std=c++11` |\n",
    "| Ubuntu 14.04 / RedHat 7 (2014-06) | 4.8                    |\n",
    "| Debian 8 (2015-04)           | 4.9                          |\n",
    "| Ubuntu 16.04                 | 5.3                          |"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "C++17",
   "language": "C++17",
   "name": "xcpp17"
  },
  "language_info": {
   "name": "C++17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
